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We consider extended Hubbard models with repulsive interactions on a Honeycomb lattice and 
the transitions from the semi-metal phase at half-filling to Mott insulating phases. In particular, 
due to the frustrating nature of the second-neighbor repulsive interactions, topological Mott phases 
displaying the quantum Hall and the quantum spin Hall effects are found for spinless and spinful 
fermion models, respectively. We present the mean-field phase diagram and consider the effects 
of fluctuations within the random phase approximation (RPA). Functional renormalization group 
analysis also show that these states can be favored over the topologically trivial Mott insulating 
states. 
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Introduction - Partly motivated by the discovery of 
the high T c superconductivity, Mott insulators have at- 
tracted great attention in recent years. Denned in a gen- 
eral sense, interactions drive a quantum phase transition 
from a metallic ground state to an insulating ground 
state in these systems. Most Mott insulators found in 
nature also have conventional order parameters, describ- 
ing, for example, the charge-density-wave (CDW) or the 
spin-density-wave (SDW) orders. However, Mott insula- 
tors with exotic ground states, such as the current carry- 
ing ground states have also been proposed theoretically 
P> S B 3< I n parallel with the study of strongly cor- 
related systems, there has recently been a growing in- 
terest in realizing topologically non-trivial states of mat- 
ter in band insulators. In the quantum anomalous Hall 
(QAH) insulators the ground state breaks time re- 
versal symmetry but does not break the lattice transla- 
tional symmetry. The ground state has a bulk insulating 
gap, but has chiral edge states. In the quantum spin 
Hall (QSH) insulators0,i,[3, the ground state does not 
break time reversal symmetry, has a bulk insulating gap, 
but has helical edge states, where electrons with the op- 
posite spins counter-propagate. The QSH state has re- 
cently been observed experimentally in HgTe quantum 
wellsBQ. 

Given the tremendous interest in finding Mott insula- 
tors with exotic ground states, and the recent discovery of 
the topologically non-trivial band insulators, it is natural 
to ask whether one can find examples of topological Mott 
insulators, which we define as states with bulk insulat- 
ing gaps driven by the interaction, and inside which lie 
topologically protected edge states. Furthermore, elec- 
tronic states in the Mott insulator phases are character- 
ized by top ological invariants, namely, the U(l) Chern 
number I 111 in the case of the QAH state, and the Z2 
invariant [121] in the case of the QSH state. In this letter, 
we report on the first example of such a case by system- 
atically studying Hubbard models with repulsive interac- 
tions on a two dimensional honeycomb lattice. In partic- 
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FIG. 1: Interactions considered in our model Hamiltonian 
(left-most hexagon), Eq. [1] Various order parameters 
are shown for the A-sublattice (open circles) in the middle 
hexagon and for the B-sublattice (filled circles) in the right- 
most hexagon. The CDW order parameter p is a real scalar, 
SDW order parameter M a real vector and the QAH/QSH 
order parameters xa,Xb are complex 4- vectors. In the case 
of spinless fermions, xa,Xb are complex scalars. xa,Xb are 
both defined on the directed second neighbor links defined by 
hi. 



ular, we demonstrate how, due to the frustrated nature 
of second-neighbor repulsion on this lattice, topological 
Mott phases displaying the QAH and the QSH effects 
are generated dynamically. We present the mean-field 
phase diagram, proceed to consider effects beyond mean- 
field theory via a functional renormalization group (fRG) 
treatment, and consider the effects arising from fluctua- 
tions. The two dimensional graphene sheet, which was 
recently realized experimentally [13l . Il4j | , contains the ba- 
sic degrees of freedom of our model. However, presently 
we do not know how to tune the interaction experimen- 
tally so that our proposed state can be realized. 



Spinless Fermions and the QAH state - The model 
Hamiltonian for spinless fermions with nearest-neighbor 
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and next-nearest neighbor interactions is written as 
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where V\ and V% are nearest-neighbor and next-nearest- 
neighbor interaction strengths, respectively. Since the 
honeycomb lattice is bipartite, consisting of two tri- 
angular sublattices (referred to here as A and B), 
nearest-neighbor repulsion will favor a charge density 
wave (CDW) phase with an order parameter p = 

\ ((^a^a) — ( c iB CiB )) that is consistent with overall 
charge conservation and describes a phase with a bro- 
ken discrete (inversion) symmetry. However, since the 
second neighbor interactions within a sublattice are frus- 
trated, CDW order will be suppressed; instead we con- 
sider the possibility of orbital ordering by defining the 
following order parameter for i,j next nearest neigh- 
bors: Xij — Xji — ( c l c j)- Let ai, 0-2,0,3 be the nearest- 
neighbor displacements from a B-site to an A-site such 
that z ■ ai x a 2 is positive. We also define the displace- 
ments bi = a 2 — a 3 , b 2 = a 3 — a%, etc, which connect 
two neighboring sites on the same sublattice (Fig. [TJ. 
A translational and rotational invariant ansatz of \ij is 
chosen as 
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which are complex scalars that live along the directed 
second neighbor links. 

Owing to translational symmetry, the mean-field free 
energy at T = is readily obtained: 
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We have defined the following quantities: 

= En=l ex P(* fc,fl »)' 4> = (<t>A + <Pb) /%,<!> = 
(<j>A ~ <i>B)/2,S k+ $ = J2 n =l sin ( k ■ b n + </>) , S<j, = sin0. 
Thus, the next-neighbor hopping amplitudes are purely 
real only when both cf) — and <f> = 0. 

When both p and \ = 0? an d at half- filling, the sys- 
tem remains a semi-metal consisting of two Fermi points 
K± which obey K± ■ bi — ±2tt/3 and the density of 
states vanishes linearly; the dispersion in the vicinity of 
these so called Dirac points is governed by a 2D massless 
Dirac Hamiltonian in fe-space . The CDW phase corre- 
sponds to an ordinary insulator with a gap at the Fermi 
energy. As for the order parameter \i which describes 
the second-neighbor hopping, its phase relative to the 
nearest neighbor hopping amplitude plays an important 



role in determining its properties: while a non-zero Re{\) 
merely shifts the energy of the Dirac points, a non-zero 
imaginary part Im(x) opens a gap at the Fermi points. 
Thus, when the system remains at half-filling, it is more 
favorable to develop purely imaginary next-neighbor hop- 
ping amplitudes; such a configuration corresponds to a 
phase with spontaneously broken time-reversal symmetry. 

To see whether such a phase can be favored, we min- 
imize the free-energy and arrive at the following self- 
consitent equations: 
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When x — 0, it is easy to see from the first equation above 
that CDW order develops continuously at a critical value 
Vic given by 



- =—y 



1 



2L 2 



\t(k) 



(6) 



Due to the vanishing density of states (DOS) near the 
Fermi points, there is no instability towards CDW for- 
mation with infinitesimal interactions. Interestingly, the 
sclf-consitent equation for x shows that a non-trivial 
self-consistent solution can only occur when (f> ^ 0; a 
detailed investigation of these equations [l5| show that 
when V\ = 0, beyond a critical value of V 2c > 0, which 
satisfies 



1 

Wc 



—y 



^k+4> 

\tk\ ' 



(7) 



a phase in which \x\ > ,<f> = 0, and <fi — ±ir/2 is favored. 
Such a phase is also stable at finite V\ and is thus does 
not require fine-tuning (see Fig. [2]). In this phase, the 
system acquires purely imaginary second-neighbor hop- 
pings with a chirality which is determined by the sign 
of <p. There is a discrete symmetry breaking correspond- 
ing to choosing <f> = ±7r/2, each of which breaks time- 
reversal symmetry. The band insulator version of the 
CDW state was considered in Ref. 16(, while the QH 



state on a honeycomb lattice was considered in Ref. [5[. 
As discussed in Ref. @ the QAH phase is a topological 
phase in which the filled states form a band which has 
a non-zero topological Chern number 11( and is an in- 



teger quantum Hall effect phase that is realized without 
Landau levels. Similar states can be constructed from 
magnetic semiconductors 0. We shall generally refer to 
QH states without Landau levels, and with full lattice 
translation symmetry as the Quantum Anomalous Hall 
(QAH) states. In our case, the topologically non-trivial 
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FIG. 2: Phase diagram for spinless fermions (t — 1). The 
semi-metallic (SM) state that occurs at weak-coupling is sep- 
arated from the CDW and the topological QAH states via a 
continuous transition (blue curve). The line separating the 
QAH and CDW marks a first-order transition (shown in red), 
which terminates at a bi-critical point. 



gap for the QAH state arises from many-body interac- 
tions rather than single particle physics, and we shall 
refer to such states as topological Mott insulators. 

We have obtained the complete phase diagram in the 
Vi ~ V% plane; within mean-field theory, there is a contin- 
uous transition from the semi-metal to either the CDW 
or the QAH phase and there is also a first-order transition 
from the CDW to the QAH phase. By integrating out 
the fermionic fields, it is possible to construct a Landau- 
Ginzburg (LG) theory expansion near the nodal region 
where the order parameters are vanishingly small. Due 
to the linear dispersion of the Fermi points, the LG free- 
energy contains anomalous terms of the form |p| 3 and 
\Im(x)\ 3 arising from linear dispersion in the vicinity of 
the Fermi points [lij]. The significance of such terms is 
that even within mean-field theory, the CDW order pa- 
rameter, for instance, grows as (Vi — V\ c ) rather than 

1/2 f — nli 

the usual {V\ — Vi c ) 17]. Furthermore, the Landau- 
Ginzburg theory describing the competition between the 
CDW and QAH phases confirms the existence of the first 
order line between these two phases that terminates in 
the bicritical point leading into the semimetal phase [l5[ . 

Spinful fermions and the QSH state - Next, we take 
into account the spin degrees of freedom and include 
an onsite Hubbard repulstion in our model Hamiltonian 
(ji = 0): 



+Vi 53(nj - l)(nj -1) + V 2 ( n * - - l ) ( 8 ) 

where = + . Since the Honeycomb lattice is bi- 
partite, onsite repulsion gives rise to a spin density wave 
phase (SDW); a standard decomposition of the Hubbard 
term introduces the order parameter M describing an 



antiferromagnetic SDW: M = ± ((S iA ) - (S lB )) ■ As 
in the spinless case, nearest-neighbor repulsion favors a 
CDW. However, there are several possible phases due 
to second-neighbor repulsion. Again, since the second- 
neighbor repulsion is frustrated, we are again led to the 
possibility of a topological phase similar to the QAH. 
However, the spin degrees of freedom introduce two pos- 
sibilities (translation invariance along with spin conserva- 
tion eliminate other possibilities): 1) two copies of QAH 
states - i.e. the chirality of the second-neighbor hopping 
is the same for each spin projection, 2) the QSH state, 
where the chiralities are opposite for each spin projection. 
The latter possibility breaks a continuous global SU (2) 
symmetry associated with choosing the spin projection 
axis; however, time-reversal symmetry is preserved. The 
QSH state on the honeycomb lattice was considered in 
Ref. 0], where the insulating gap arises from the micro- 
scopic spin-orbit coupling. It was shown latter that the 
magnitude of of the spin-orbit gap is negligibly small in 
graphene [3, 0|. In our case, the insulating gap is gen- 
erated dynamically from the many-body interaction. In 
this sense, our effect can be viewed as an example of dy- 
namic generation of spin-orbit interaction[20]. Introduc- 
ing the Hubbard-Strataovich fields (sum over repeated 

indices implied) x% = c \a a ap c iP^ A 4 = 0...3, where 
(7^ = (1,<t), the next-neighbor interactions can be re- 
cast using the identity (n, - l){n } - 1) = 1- | (x y ) X%- 
Physically, if (x°) ^ 0, then we are in the QAH phase. 
If, on the other hand, one of the vector components 
(X*) 7^ 0' then we are in the QSH phase. A transla- 
tionally invariant decomposition of the next-neighbor in- 
teractions via (Xii+b ) = X^e 1 ^- 4 :* £ ^ (and similarly 
for the other sublattice) gives rise to a 4 x 4 Hamiltonian 
is readily diagonalized in a tensor product basis cr (g> t, 
where cr and r are Pauli matrices in spin and sublattice 
space, respectively. This way, each phase corresponds to 
a particular non-zero expectation value of a fermion bi- 
linear J^k *£^(^)*jfc' wnere d(k) oc t 3 for the CDW and 
QAH, d{k) oc cr 3 r 3 for SDW and QSH. A detailed study 
of the free-energy at T — and its saddle point solu- 
tions [IH produces the phase diagram shown in Fig. [3] 
In addition to the ordinary CDW and SDW insulating 
phases, there is a phase for V% > V% c ~ l-2t in which 
the 4- vector is purely imaginary (as in the spinless case), 
collinear, and staggered from one sublattice to the next: 

(Xi i+bn , A ) = -(Xii+b n ,B)> and both Q AH and Q SH are 
equally favorable ground states, having identical free en- 
ergies within mean-field theory. Additionally, there is 
never a coexistence of both QAH and QSH phases; in- 
deed, a Landau- Ginzburg treatment in this region explic- 
itly shows the absence 50(4) symmetry of the vector x M - 
This occurs due to the difference of the manner in which 
X° and x are coupled to the fermionic fields - which favors 
either a phase with broken Zi symmetry (QAH) or with 
broken SU (2) symmetry, but never both simultaneously 
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FIG. 3: Complete mean-field phase diagram for the spinful 
model. The transitions from the semimetal (SM) to the in- 
sulating phases are continuous, whereas transitions between 
any two insulating phases (red lines) are first-order. 




FIG. 4: a) Data for [7=0, Vi=1.4i, V 2 =0. Susceptibilities of 
each phase vs. T are shown: CDW (black); SDW (green); 
QAH(red) and QSH (blue).6j Same for 17=0, Vi=0, V 2 =IM 
(QSH instability). The QSH phase has a larger susceptibility 
than QAH. Inset: fRG phase diagram at U—O, indicating SM 
(blue) and insulating (red) regions (CDW dominates at large 
Vi, QSH at large V2 ) The colorbar correspond to T c below 
which the insulating phases develop in fRG. 



[151. 

Quantum fluctuations, however, lift the mean-field 
degeneracy between the QAH and QSH phases. To 
quadratic order in quantum fluctuations (RPA) about 
the QSH phase , we obtain an effective action S e ff — 
Y,k 5 X^{k,VL)K^{k,n)5 X v {-k,-VL) which shows the 
presence of six modes (2 longitudinal and 4 transverse 
modes) , and 2 of the transverse modes correspond to de- 
generate Goldstone modes whose velocity is proportion- 
ality to the Fermi velocity v ~ Vf = 3t/2\a\. Thus, the 
zero-point motion associated with these gapless modes, 
lowers the free energy of the QSH state relative to the 
QAH state. 

Renormalization Group Analysis - Mean field theory 
generally starts with a given, in a sense, biased Ansatz, 
and investigate the self-consistency of the mean field so- 
lution. Therefore, it is important to investigate the topo- 
logical Mott states with a method without any a priori 
bias. Next we go beyond mean-field theory and RPA us- 
ing the temperature(T)-fiow functional renormalization 
group (fRG) [21] 22]. In this scheme, we discretize the k- 
dependence of the interaction Q and consider all possi- 
ble scattering processes between a set of initial and final 
momenta that occur between points on rings around the 
Dirac points (inset of Fig. 0J. Starting with T ~ 2t, 
the temperature T is lowered, and a flowing (renormal- 
ized) interaction Vt is obtained by the coupled summa- 
tion of the T-derivatives of all one-loop channels. Due 
to this, the method is unbiased and goes beyond the 
mean-field-level. Applying the scheme to the Hamilto- 
nian, Eq. [8l we search for flows to strong coupling, where 
for a low temperature T c certain components of Vt be- 
come large. Then the approximations break down, and 
the flow is stopped. Information on the low-T state is 
obtained from analyzing which coupling functions grow 
most strongly and from susceptibilities for static external 



fields coupling to the various order parameters. In this 
scheme, a tendency towards ordering at a finite vector 
Q can be detected as a growth of the associated vertex 
Vt- However, we have found that largest couplings occur 
at Q = 0, which strongly supports the mean-field results 
presented above. 

For onsite and nearest-neighbor repulsions U > U c ~ 
3.8t and V\ > V\ c ~ 1.2i, the flow to strong coupling 
is either an SDW instability for dominant U or a CDW 
instability for dominant V\, in good agreement with a 
l/N-studv[24j and Quantum-Monte-Carlo [l7|. For more 
details, see Ref. (25|. If we include a sufficiently strong 
second-nearest-neighbor repulsion V2 > 1.6t, the flows 
change qualitatively; there is a leading growth of the QSH 
susceptibility. In Fig. 2]a) and b) we compare the T-flows 
of various susceptibilities for V\ > V2 and for V2 > V\ . 
For the latter case, the QSH susceptibility grows most 
strongly toward low T, followed by the QAH susceptibil- 
ity, which is consistent with the RPA treatment of the 
Goldstone modes in the QSH. The QSH phase remains 
stable even when a moderate onsite interaction of U = t 
or U = 2t is introduced. Hence the global structure of 
the mean-field phase diagram is confirmed by the fRG 
results. Note however that the slope of the lines of criti- 
cal V\ versus V2 differs. We interpret this a competition 
effect captured by the fRG, where V% decreases the CDW 
tendencies induced by V\ . 

Discussion - We have shown that topological phases 
displaying the QAH and QSH effects can be generated 
from strong interactions - thus, we refer to these phases 
as topological Mott insulators. Both phases have asso- 
ciated with them conventional order parameters which 
develop continuously at the quantum critical phase tran- 
sition from the semi-metallic state. However, these states 
are also described by topological quantum numbers which 
jump discontinuously at the transition. Although the in- 
teraction strengths needed to produce these phases are 
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strong, we expect that our mean-field, RPA and fRG 
treatment to provide strong evidence for the existence of 
a topological Mott insulator: the Stoner criterion, which 
states that perturbation theory breaks down when in- 
teraction strengths are comparable to the inverse of the 
effective DOS, suggests that perturbation theory is more 
robust in our system due to the vanishing of the DOS at 
the Fermi points. An open issue remains to find other 
possible realizations of the phases described here. Fur- 
thermore, the nature of the low energy effective coupling 
of the gapless bulk Goldstone modes with the gapless 
edge degrees of freedom, and the possibility of fractional- 
ized excitations in these phases are interesting open issues 
that we leave for future work. 
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